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ABSTRACT 

The  proposed  technology  is  a  three  dimensional  (3D) 
and  sensor  fusion  display  for  threat  and  battlefield 
damage  assessment  (BDA).  The  3D  auto-stereoscopic 
display  is  implemented  for  Homeland  defense 
applications  in  combination  with  an  X-ray  scanner  that 
will  bring  attention  to  packages  with  potential  threats. 
Sensor  fusion  will  benefit  BDA  by  providing  fused  visible 
and  infrared  (IR)  images  of  combat  vehicles  that  have 
been  damaged  or  destroyed  on  the  battlefield. 


1.  INTRODUCTION 

Battle  Damage  Assessment  is  a  requirement  for 
current  and  future  ground  vehicle  missions.  Three 
subcomponents  of  BDA  are  physical  damage  assessment 
(PDA),  functional  damage  assessment  (FDA)  and  target 
system  assessment  (TSA).  Our  proposed  technical 
approach  is  to  provide  a  visual  evaluation  for  the 
effectiveness  of  the  target’s  capability  with  surveillance 
systems.  The  advanced  BDA  will  increase  the 
survivability  and  lethality  of  combat  systems,  and  reduce 
the  precision  munitions  expenditure.  The  proposed 
concept  system  would  include  a  fused  image  of  visible 
and  IR  combined  with  a  real-time  3D  display  that  will  be 
tested  under  field  conditions.  The  system  will  support 
real-time  BDA  by  providing  fused  visible  and  IR  pictures 
of  combat  vehicles.  This  assessment  will  allow  soldiers 
to  identify  combat  vehicles  that  have  been  damaged  or 
destroyed  on  the  battlefield  and  eventually  distinguish 
civilians  from  insurgents.  The  task  will  be  accomplished 
using  the  sensor  fusion  technique  for  visual  and  IR 
imagery.  The  battle  damage  assessment  with  3D  depth 
perception  will  be  accomplished  by  using  multiple  camera 
views  displayed  directly  using  an  auto-stereoscopic 
display.  This  technology  was  tested  in  the  lab 
environment  for  threat  recognition  and  camouflage 
assessment.  Sensing,  assessing,  and  reporting  the  success 
or  failure  of  battle  damage  (e.g.  disabled,  damaged, 
destroyed)  of  combat  vehicles  and  targets  is  essential  for 
successful  mission  accomplishment.  Post-strike  BDA 
must  be  reported  to  the  commanding  authority  for 
evaluation  and  response  decision,  such  as  re-attack.  The 
proposed  technology  will  expand  the  warfighter’s 
perception,  situational  awareness  (SA),  and  knowledge  of 
the  battlefield  situation  beyond  the  limits  that  were 


available  through  visual  and  other  sensory  perception. 
Warfighter  payoff  is  the  real  time  damage  assessment  of 
combat  vehicles,  dynamic  surveillance,  data  fusion  and 
3D  visualization. 


2.  THREAT  ASSESSMENT  USING  3D 
DISPLAY 


It  has  been  estimated  that  failures  in  advanced  man- 
machine  systems  attributed  to  “human  error”,  and 
increasingly  to  loss  of  operator  SA,  and  may  be  as  high  as 
75%.  Three  dimensional  displays  will  increase  SA  by 
allowing  a  user  to  see  information  collected  by  sensors  in 
a  natural  environment  as  opposed  to  planar  images.  There 
are  a  number  of  mirror  systems  which  form  a  real  three 
dimensional  image  in  space  of  one  object  in  a  position 
which  is  well  separated  from  another  object.  One  of  the 
major  parts  of  the  stereoscopic  display  used  in  the  VPL  is 
the  spherical  concave  mirror.  A  spherical  mirror  can 
magnify  the  size  of  the  object  being  displayed.  The 
paraxial  equation  for  a  spherical  mirror  is  shown  below  in 
equation  (1): 
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Here,  s  is  the  object  distance  from  the  mirror,  s'  is 
the  image  distance,  r  is  the  radius  of  curvature  of  the 
mirror,  and  f  is  the  focal  length  of  this  mirror.  The 
magnification  m  can  be  defined  as  the  ratio  of  the  image 
size  to  the  object  size  as  shown  in  equation  (2): 
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Here,  h'  is  the  image  size,  and  h  is  the  object  size. 
The  magnification  can  also  be  defined  in  terms  of  the 
image  distance  and  the  object  distance.  The  minus  sign 
indicates  that  the  image  is  inverted  (Ditteon,  1997).  A 
magnified  image  is  obtained  with  the  object  distance 


1 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

00  DEC  2004 

2.  REPORT  TYPE 

N/A 

3.  DATES  COVERED 

4.  TITLE  AND  SUBTITLE 

5a.  CONTRACT  NUMBER 

Threat  And  Battlefield  Damage  Assessment  Using  3d  Imaging  And 

5b.  GRANT  NUMBER 

ocnaui  r  usiuit 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  RDECOM-Visual  Perception  Laboratory  (VPL)  Warren,  MI 
48397 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release,  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

See  also  ADM001736,  Proceedings  for  the  Army  Science  Conference  (24th)  Held  on  29  November  -  2 
December  2004  in  Orlando,  Florida.,  The  original  document  contains  color  images. 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

uu 

18.  NUMBER 
OF  PAGES 

4 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


between  the  focal  length  and  the  center  of  curvature  of  the 
mirror.  The  new  3D  display  system  with  a  solid  fixed 
focal  length  mirror  used  for  Homeland  defense 
applications  provides  magnification  up  to  two  times  the 
image  size. 

The  Visual  Perception  Lab  (VPL)  at  RDECOM- 
TARDEC  proposed  the  3D  auto-stereoscopic  display  for 
various  defense  applications.  This  3D  display  uses 
spherical  membrane  mirror  technology.  This  technology 
was  developed  by  engineers  and  scientists  of  the 
Department  of  Mechanical  Engineering  at  the  University 
of  Strathclyde,  in  Glasgow,  UK  (McKay  et  al.,  1999). 
The  spherical  membrane  mirror  was  later  implemented  in 
the  auto-stereoscopic  3D  display  by  Ethereal 
Technologies.  A  vacuum  pump  is  used  to  shape  the 
mirror  membrane  to  the  desired  spherical  shape. 
Researchers  in  the  VPL  use  the  large  field  of  view 
Ethereal  display  for  feasibility  studies  using  3D.  The 
images  for  the  3D  auto-stereoscopic  display  can  be 
captured  by  a  stereoscopic  camera,  or  by  shifting  2D 
images  in  space,  or  combining  pairs  of  images  taken  at 
different  times.  The  pairs  of  images  are  downloaded  into 
the  computer,  and  displayed  on  two  LCD  screens.  The 
left  view  image  is  displayed  on  the  right  hand  LCD,  and 
the  right  view  image  is  displayed  on  the  left  hand  LCD. 
These  images  are  inverted  by  the  lens  and  mirror  system 
shown  in  Figure  1.  Another  possible  application  of  this 
technology  in  some  rugged  form  is  a  lenticular  display  for 
use  in  vehicles. 


Observer 


Figure  1:  VPL  auto-stereoscopic  3D  display 

In  asymmetrical  warfare,  the  civilians  are  considered 
targets.  There  is  a  need  for  technology  to  scan  packages 
and  use  this  3D  system  to  reduce  illegal  transport  of 
weapons  and  potential  threats  on  U.S.  soil.  This  display 
can  be  applied  to  crowd  surveillance  and  screening  at 


public  access  points.  In  these  times  of  increased  concerns 
about  terrorists  and  passengers  carrying  concealed 
weapons  or  other  harmful  items,  sensor  fusion  and  3D 
displays  could  be  of  benefit  in  alerting  guards  to  potential 
dangerous  passengers.  Combining  sensor  fusion  with  a 
3D  display  could  also  improve  the  recognition  rate  of 
guards  using  cameras  that  scan  crowds  for  people  that  are 
listed  in  a  known  terrorist  database.  We  believe  that 
implementing  volumetric  visible  and  IR  images  will 
improve  the  recognition  rate,  because  it  will  provide  a 
more  detailed  3D  image  or  thermal  properties  versus  only 
a  2D  visible  image  of  possible  terrorization  (Meitzler,  et 
al.,  2002). 

Another  important  application  of  a  3D  display  is 
improved  visualization  of  potential  threats  during  package 
scanning.  Professor  Bob  Andrews  from  Ethereal 
Technologies  is  presently  working  with  the 
Transportation  Security  Authority  (TSA)  in  Atlantic  City 
and  the  VPL  at  RDECOM  on  implementing  the  auto- 
stereoscopic  3D  display  technology  for  X-ray  package 
scanning.  A  set  of  images  was  collected  from  the  X-ray 
scanner  at  the  US  Army  RDECOM  shipping  and 
receiving  department.  These  images  were  then  converted 
into  stereoscopic  pairs  to  be  displayed  on  the  auto- 
stereoscopic  3D  display. 

Today,  safety  is  implemented  by  not  allowing 
the  RDECOM  employee  to  open  any  shipment  packages 
without  examining  it  first.  A  properly  trained  RDECOM 
employee  must  scan  the  package  by  using  an  X-ray 
scanner,  where  the  image  is  displayed  on  the  monitor 
which  provides  a  2D  image  of  the  objects  inside  the 
package.  As  seen  in  Figure  2  the  color  depends  on  the 
density  of  the  material  or  object  in  the  package.  For 
instance,  a  bluish  or  orange  color  represents  denser  or  less 
dense  materials  respectively.  The  test  image  of  the  faux 
bomb  from  the  shipping  and  receiving  department  X-ray 
scanner  is  shown  in  Figure  2.  The  image  was  provided  by 
Mr.  Leonard  Miles,  a  contractor  at  RDECOM  shipping 
and  receiving  department.  This  inert  bomb  was  used  to 
test  employees  on  their  inspection  skills.  The  2D  X-ray 
scan  shown  in  Figure  2  does  not  provide  the  screener  with 
the  depth  perception  of  the  object  which  would  have 
made  it  easier  for  them  to  find  harmful  items. 


Figure  2:  The  test  image  of  a  mock  bomb  from  the 
RDECOM  shipping  and  receiving  dock  X-ray  scanner 
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The  information  that  was  received  regarding  the 
tactics  of  the  X-ray  scanner  was  further  tested  in 
correlation  with  the  3D  display  mirror  in  the  VPL 
laboratory.  Figure  3  provides  an  example  of  a  real  set  of 
pictures  of  a  package  that  was  collected  from  the  X-ray 
scanner.  To  create  3D  images  from  the  X-ray  scanner 
VPL  team  members  captured  pictures  of  questionable 
packages  at  different  angles  in  one  plane.  The  images 
were  then  placed  into  a  stereoscopic  format  for  the  3D 
display  that  would  then  convert  this  same  image  into  3D. 
The  left  image  is  a  plain  image  of  a  package  inside  the  X- 
ray  scanner.  The  right  image  was  obtained  by  raising  the 
same  package  in  the  z  plane  with  a  piece  of  Styrofoam, 
which  is  invisible  to  the  scanner.  The  x  and  y  coordinates 
of  the  package  in  the  right  image  were  the  same  as  in  the 
left  image.  As  seen  in  Figure  3  the  left  and  right  eye 
images  are  distorted  somewhat  from  each  other.  These 
images  were  converted  into  stereoscopic  pairs  to  be 
displayed  using  the  auto-stereoscopic  3D  display  mirror 
in  the  VPL.  This  gives  a  person  depth  perception  and 
may  increase  detection  of  dangerous  material  varying 
from  different  areas  whether  in  an  airport  or  at  a  place  of 
business. 
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3.  THREAT  ASSESSMENT  USING  IMAGE 
FUSION 

Another  component  of  the  proposed  technology 
for  BDA  is  image  fusion  for  Homeland  defense 
applications  in  which  RDECOM’s  VPL  performed  a  test 
using  a  rocket  propelled  grenade  (RPG).  Image  sets  are 
of  a  warfighter  holding  a  RPG  that  were  composed  of  an 
IR,  visible  and  a  fused  image  arranged  side  by  side  as 
shown  in  Figure  4  below.  A  quality  rating  of  the  RPG 
images  was  given  to  determine  which  of  the  three  images 
provided  more  detail.  We  asked  test  subjects  to  rate 
images  on  a  scale  from  1  to  3,  with  1  being  the  easiest  to 
detect  the  RPG.  Although  the  RPG  was  easily  visible  in 
the  IR  image  and  most  fused  images,  it  was  not  readily 
apparent  in  the  visible  images.  Due  to  sometimes  poor 
quality  of  the  visible  images,  the  fused  images  would 
suffer  as  well,  since  they  were  constructed  from  the 
visible  and  IR  images.  Future  work  could  strive  to 
develop  a  smarter  enhancement  algorithm  for  image 
fusion. 


(a)  IR  image;  (b)  Visible  image;  (c)  Fused  images 


Figure  4:  Images  captured  of  a  warfighter  holding  a  RPG 
by  using  different  image  systems 


(a)  Left  eye  image;  (b)  Right  eye  image 

Figure  3:  The  stereoscopic  pair  of  images  from 
RDECOM’s  shipping  and  receiving  dock 

Viewing  this  pair  of  images  on  the  3D  auto- 
stereoscopic  display  shown  in  Figure  3  gives  a  better 
depth  perception  of  a  package.  The  3D  representation 
will  assist  the  screener  in  identifying  what  the  object 
really  is.  It  will  allow  the  screener  to  see  the  information 
collected  by  the  X-ray  machine  in  a  natural  3D 
environment  as  opposed  to  the  planar  images.  The  future 
systems  will  implement  sensor  fusion  combined  with  the 
3D  visualization.  The  images  will  be  fused  digitally  to 
present  them  through  stereopsis.  Stereopsis  is  the  process 
of  using  our  human  visual  system  to  extract  depth 
information  to  build  a  3D  understanding  of  a  scene. 
Some  people  who  cannot  experience  stereopsis  will  see  a 
flat  image,  rather  than  a  stereoscopic  image.  The  VPL  is 
planning  to  do  3D  assessment  tests  that  compare  images 
in  2D  to  3D  of  response  time  and  false  alarm  rates.  A  3D 
test  can  continue  once  it  is  determined  that  the  test  subject 
can  experience  depth  perception. 


The  statistical  results  of  the  RPG  test  were  analyzed 
as  seen  below  in,  Figure  5,  which  shows  the  subjective 
quality  rate  of  the  RPG  in  the  image  versus  the  sensor 
type. 


The  Quality  Rating  of  Various  Systems 


Cameras/Sensors 


Figure  5:  The  quality  rating  of  various  cameras  and 
sensor  systems  as  seen  in  Figure  4  images 
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This  graph  illustrates  that  the  highest  detection  rate 
was  achieved  using  the  IR  sensor,  and  the  lowest 
detection  rate  was  found  with  the  visible  image.  Even 
though  the  test  subjects  chose  the  IR  imagery  over  the 
fused  for  seeing  the  RPG  weapon,  they  commented  on  the 
fact  that  they  preferred  the  fused  image,  because  it 
provided  more  details  to  the  area  of  interest. 

The  sensor  fusion  technique  can  benefit  the  post¬ 
strike  BDA  activity,  because  it  can  obtain  information 
about  combat  vehicles  that  have  been  damaged  or 
destroyed  on  the  battlefield  using  IR  images.  However,  a 
fused  image  of  the  battlefield  will  provide  more  detail, 
because  it  has  a  wider  spectral  band.  The  spectral  band 
for  visible  light  ranges  from  0.4  to  0.7  microns.  The 
FLIR  FSI  digital  automatic  camera  that  we  used  for 
imaging  in  the  IR  part  of  the  spectrum  had  light 
wavelength  ranging  from  2  to  8  microns.  The  fused 
image  had  the  combined  spectral  range  of  both  visible  and 
IR  cameras.  As  an  illustration  of  this  fact,  Figure  6  (a) 
shows  an  IR  image  of  an  automobile  in  a  parking  lot,  (b) 
shows  a  visible  image  of  the  same  scene,  and  (c)  shows 
the  fused  IR  and  visible  image  of  the  scene.  The  fused 
image  was  obtained  using  a  customized,  in-house  Matlab 
program.  Similar  to  the  RPG  test,  the  improved  quality  of 
the  image  in  Figure  6  (c)  can  be  easily  observed. 


(a)  IR  image;  (b)  Visible  image; 


(c)  Fused  IR  and  visible  image 


Figure  6:  Images  of  a  scene  with  an  automobile,  captured 
using  different  imaging  systems 


CONCLUSIONS 


vehicles  with  IR  signatures.  It  will  enable  identification 
of  vehicles  that  have  been  damaged  or  destroyed.  The 
battlefield  could  be  accessed  promptly  using  a  3D  display 
with  depth  perception  capability  for  accurate  BDA.  It 
will  expand  the  warfighter’s  perception,  situational 
awareness  and  knowledge  of  the  battlefield  situation 
beyond  the  limits  that  were  available  through  visual  and 
other  sensory  perception.  New  IR  cameras  for  defense 
applications  are  being  developed  by  the  FLIR 
Corporation.  In  the  future  we  would  like  to  evaluate  the 
AF/256  super  sensitive  thermal  imaging  system 
developed  by  FLIR  Corporation.  This  system  is  ideal  for 
medium  to  long  range  detection.  The  unit  has  the 
capabilities  of  detecting  a  human  up  to  8  Kilometers  in 
total  darkness  and  in  adverse  conditions.  The  heart  and 
soul  of  the  AF/256  is  an  Indium  antimonide  infrared 
detector  which  offers  ultra  sensitive  imaging  and  is 
widely  utilized  by  the  United  States  and  NATO  countries. 
This  is  an  ultimate  system  for  BDA,  airfield,  harbor  and 
critical  installation  surveillance.  The  proposed  new 
technologies  will  utilize  joint  intelligence  estimates 
provided  by  a  layered  sensors  approach  and  will  develop 
BDA  to  support  all  fire  missions  and  joint  assets. 
Warfighter  payoff  will  be  the  real  time  damage 
assessment  of  combat  vehicles,  dynamic  surveillance, 
image  based  sensor  fusion  and  3D  visualization.  Another 
important  application  of  a  3D  display  technology  for 
Homeland  defense  is  improved  visualization  of  potential 
threats  during  package  scanning.  By  using  the  3D  display 
technology  the  screeners  sees  depth  in  the  object.  It  will 
allow  the  screener  to  see  the  information  collected  by  the 
X-ray  machine  in  a  natural  3D  environment  as  opposed  to 
the  planar  images.  In  the  future  work  we  would  like  to 
investigate  the  experiments  with  sensor  fusion  further  to 
include  magnetic  sensors,  which  will  facilitate  more 
accurate  BDA.  The  future  systems  will  implement  sensor 
fusion  combined  with  the  3D  visualization. 
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Maintaining  situational  awareness,  dynamic 
surveillance,  and  target  development  is  important  for  the 
warfighter.  This  task  will  be  accomplished  through 
sensor  acquisition,  data  fusion  and  3D  visualization.  The 
implementation  of  the  integrated  image  system  will 
support  real-time  BDA  by  providing  images  of  combat 
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